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IMAGE -PROCESSING COMPUTER SYSTEM 
FOR PHOTOGRAMMETRIC ANALYTICAL MEASUREMENT 



BACKGROUND OF THE INVENTION 
5 1 . Field of the Invention. 

The present invention relates to an image-processing 
computer system for a photogrammetric analytical measurement, 
in which a survey map is electronically produced based on a 
pair of photographed pictures obtained at two different 

10 photographing positions . 

2 . Description of the Related Art 

For example, photogrammetry is carried out at a traffic 
accident spot. The traffic accident spot is photographed by 
an electronic still video digital camera in at least two 

15 different positions, and a survey map of the traffic accident 
spot is produced based on a pair of photographed pictures 
obtained at the different positions, as disclosed in, for 
example, Unexamined Japanese Patent Publications No. 10-221072 
and No. 10-293026. 

20 Before accurately scaled distances and lengths can be 

reproduced on the survey map, a standard measurement scale or 
target must be recorded together with the photographed objects 
in the pictures. The target is disclosed in, for example, 
Unexamined Japanese Patent Publications No . 10-141 951 , 

25 No . 10-170263 , No . 10-185562 , No . 10-185563 , No. 10-293026 and 
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No . 10-307025 . 

For the production of the survey map, a two-dimensional 
coordinate system is defined on each of the pictures, and 
two-dimensional positions of the objects, which are recorded 
5 on each picture, are determined by the two-dimensional 
coordinate system. Also, a three-dimensional coordinate 
system is defined on the target, and three-dimensional 
positions of the recorded objects are calculated based on the 
two-dimensional positions of the objects with respect to the 

10 three-dimensional coordinate system. Thus, it is possible to 
produce a survey map by projecting the three-dimensional 
coordinates, representing the objects, on one of the three 
planes defined by the three-dimensional system. Of course, the 
production of the survey map is performed using an image- 

15 processing computer system. 

However, before the calculation of the three- 
dimensional positions of the recorded objects can be performed, 
it is necessary to determine camera parameters, by which the 
photographing position of the camera is represented with 

20 respect to the three-dimensional coordinate system. 

Conventionally, the determination of the camera 
parameters is performed by correspondingly indicating 
reference points of the target on scenes of the pictures, 
displayed on a monitor, by clicking on them with the mouse. 

25 This manual indication of the reference points of the target 
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is unreliable. Therefore, unless the manual indication is 
precisely performed, the determination of the camera 
parameters is inaccurate, and an accurate proper production 
of the survey map cannot be ensured. 
5 SUMMARY OF THE INVENTION 

Therefore, an object of this invention is to provide 
an image processing computer system for a photogrammetric 
analytical measurement, in which camera parameters necessary 
for production of a survey map can be automatically and 

10 precisely determined by suitably processing the image data of 
a photographed picture . 

In accordance with a first aspect of the present 
invention, there is provided an image processing computer 
system for a photogrammetric analytical measurement, in which 

15 camera parameters necessary for production of a survey map are 
determined based on a picture having an image of a target. The 
camera parameters represent a photographing position and a 
photographing direction of a camera, by which the picture is 
photographed, and the target has at least three main reference 

20 point areas and at least one assistant reference point area, 
each of the reference point areas being formed as a high 
luminance point area surrounded by a low luminance area. The 
image processing computer system comprises : a target-image 
extractor that extracts the image of the target from the picture 

25 based on positional relationships between the main and 
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assistant reference point areas of the target; an image 
processor that processes the extracted image of the target to 
determine a two-dimensional position of each of the main and 
assistant reference point areas of the target with respect to 
5 a two-dimensional picture coordinate system defined on the 
image; a first calculator that calculates three-dimensional 
positions of the main reference point areas with respect to 
a three-dimensional camera coordinate system defined on the 
camera; a second calculator that calculates two-dimensional 

10 positions of the main reference point areas with respect to 
a two-dimensional image-plane coordinate system, defined on 
an image-plane of the camera, based on the three-dimensional 
positions of the main reference point areas calculated by the 
first calculator; and a third calculator that calculates camera 

15 parameters based on the two-dimensional positions of the main 
reference point areas with respect to the two-dimensional 
picture coordinate system and the two-dimensional positions 
of the main reference point areas with respect to the two- 
dimensional image-plane coordinate system. 

20 The image processing computer system may further 

comprises: a fourth calculator that calculates a three- 
dimensional position of the assistant reference point area with 
respect to the three-dimensional camera coordinate system 
based on the camera parameters calculated by the third 

25 calculator; a fifth calculator that calculates a two- 
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dimensional position of the assistant reference point area with 
respect to the two-dimensional image-plane coordinate system 
based on the three-dimensional position of the assistant 
reference point area calculated by the fourth calculator; and 
5 a determiner that determines whether the calculation of the 
camera parameters by the third calculator is correct or 
incorrect by comparing the two-dimensional position obtained 
by the image processor with the two-dimensional position of 
the assistant reference point area calculated by the fifth 

10 calculator. 

In accordance with a second aspect of the present 
invention, there is provided an image processing method for 
a photogrammetric analytical measurement, in which camera 
parameters necessary for production of a survey map are 

15 determined based on a picture having an image of a target. The 
camera parameters represent a photographing position and a 
photographing direction of a camera, by which the picture is 
photographed, and the target has at least three main reference 
point areas and at least one assistant reference point area, 

20 each of the reference point areas being formed as a high 

luminance point area surrounded by a low luminance area . The 
image processing method comprises steps of: extracting the 
image of the target from the picture based on positional 
relationships between the main and assistant reference point 

25 areas of the target; processing the extracted image of the 
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target to determine a two-dimensional position of each of the 
main and assistant reference point areas of the target with 
respect to a two-dimensional picture coordinate system defined 
on the target; calculating three-dimensional positions of the 
5 main reference point areas with respect to a three-dimensional 
camera coordinate system defined on the camera; calculating 
two-dimensional positions of the main reference point areas 
with respect to a two-dimensional image-plane coordinate 
system, defined on an image-plane of the camera, based on the 

10 three-dimensional positions of the main reference point areas; 
and calculating camera parameters based on the two-dimensional 
positions of the main reference point areas with respect to 
the two-dimensional picture coordinate system and the two- 
dimensional positions of the main reference point areas with 

15 respect to the two-dimensional image-plane coordinate system. 

The image processing method may further comprises the 
steps of: calculating a three-dimensional position of the 
assistant reference point area with respect to the three- 
dimensional camera coordinate system based on the calculated 

20 camera parameters; calculating a two-dimensional position of 
the assistant reference point area with respect to the 
two-dimensional image-plane coordinate system based on the 
calculated three- dimensional position of the assistant 
reference point area; and determining whether the calculation 

2 5 of the camera parameters is correct or incorrect by comparing 
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the two-dimensional position based on the two-dimensional 
picture coordinate system with the two-dimensional position 
of the assistant reference point area based on the two- 
dimensional image-plane coordinate system. 
5 In accordance with a third aspect of the present 

invention, there is provided a memory medium storing an image 
processing program in which the aforesaid image processing 
method is performed. 
BRIEF DESCRIPTION OF THE DRAWING 

10 The object and other objects of the present invention 

will be better understood from the following description, with 
reference to the accompanying drawings , in which : 

Figure 1 is a plan view of a roadway for explaining a 
photogrammetric analytical measurement, which is utilized by 

15 a photogrammetric image processing computer system according 
to the present invention; 

Figure 2 is a view showing a first scene of a first 
picture photographed at a first photographing position shown 
in Fig . 1 ; 

20 Figure 3 is a view showing a second scene of a second 

picture photographed at a second photographing position shown 
in Fig . 1 ; 

Figure 4 is a perspective view of a target photographed 
on the pictures shown in Figs . 2 and 3 ; 
25 Figure 5 is a block diagram of the photogrammetric image 
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processing computer system according to the present invention; 

Figure 6 is a conceptual perspective view showing a 
positional relationship between the target and a camera 
utilized in the photogrammetric analytical measurement ; 
5 Figure 7 is a view showing a map-production-mode scene 

displayed on a TV monitor forming a part of the photogrammetric 
image processing computer system shown in Fig. 5 ; 

Figure 8 is a flowchart of a camera-parameter- 
determination routine executed in the photogrammetric image 
10 processing computer system shown in Fig. 5 ; 

Figure 9 is a flowchart of a circumscribed-rectangle- 
nomination subroutine executed in step S200 of the camera- 
parameter-determination routine of Fig. 8; 

Figure 10 is a conceptual view showing the first picture 
15 together with a reduced picture thereof, for explaining a 
production of a rectangle circumscribed about an mage of the 
target on the first pictures; 

Figure 11 is a histogram based on image pixels included 
in the reduced picture shown in Fig. 10; 
20 Figure 12 is a flowchart of a threshold-nomination 

subroutine executed in step S300 of the camera-parameter- 
determination routine of Fig . 8 ; 

Figure 13 is a histogram based on image pixels 
encompassed by the circumscribed-rectangle defined on the 
25 first picture of Fig. 10; 



Figure 14 is a flowchart of a provisional- target- 
image-determination subroutine executed in step S400 of the 
camera-parameter-determination routine of Fig. 8; 

Figure 15 is a flowchart of a provisional -balance- 
5 center-determination subroutine executed in step S4100 of the 
provisional-target-image-determination subroutine of Fig. 14; 

Figure 16 is a flowchart of a balance-center-alignment- 
detection subroutine executed in step S4200 of the 
provisional-target-image-determination subroutine of Fig. 14; 
10 Figure 17 is a flowchart of a provisional-target- 

image-confirmation subroutine executed in step S4300 of the 
provisional-target-image-determination subroutine of Fig. 14; 

Figure 18 is a flowchart of a reference-point- 
coordinate-correction subroutine executed in step S500 of the 
15 camera-parameter-determination routine of Fig. 8; 

Figure 19 is a conceptual view showing an image-pixel 
area extracted from the first picture so as to include an image 
of a reference point of the target ; 

Figure 20 is a histogram based on image pixels included 
20 in the image-pixel area shown in Fig. 19; 

Figure 21 is a conceptual view showing an 8-neighbor 
Laplacian filter used in the Laplacian-f iltering processing; 

Figure 22 is a graph for explaining the Laplacian- 
f iltering processing; 
25 Figure 23 is a flowchart of a precise-balance- 
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center-determination subroutine executed in step S5100 of the 
reference-point-coordinate-correction subroutine of Fig. 18; 

Figure 24 is a flowchart of a balance-center-selection 
subroutine executed in step S5200 of the ref erence-point- 
5 coordinate-correction subroutine of Fig. 18; 

Figure 25 is a flowchart of a camera-parameter- 
calculation subroutine executed in step S600 of the 
camera-parameter-determination routine of Fig. 8; and 

Figure 2 6 is a flowchart of a calculation-confirmation 
10 subroutine executed in step S700 of the camera-parameter- 
determination routine of Fig. 8. 
DESCRIPTION OF THE PREFERRED EMBODIMENTS 

With reference to Fig. 1, a part of a roadway is shown 
as a plan view, by way of example, for explaining a principle 
15 of a photogrammetric analytical measurement which utilizes a 
photogrammetric image processing computer according to the 
present invention. 

In this example, the roadway is photographed by an 
electronic still video digital camera 10 at two different 
20 photographing positions, which are represented by small solid 
circles indicated by references Ml and M2 , respectively. Each 
of the pictures, photographed at the photographing positions 
Ml and M2 , is processed by a control circuit of the digital 
camera 10, and is then stored as a frame of image data in a 
2 5 memory medium, such as an IC memory card, held in the camera 
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10 . The memory card is loaded in the photogrammetric image 
processing computer system according to the present invention, 
and a survey map of the roadway is produced on the basis of 
the two frames of image data read from the memory card. 
5 Each of the photographing positions Ml and M2 is defined 

as a back principal point of a photographing optical lens system 
of the digital camera 10. As shown in Fig. 1, an arrow is 
projected from each of the photographing positions Ml and M2 
to indicate a photographing direction at the corresponding 
10 photographing position (Ml, M2) . The photographing direction 
is defined as the orientation of an optical axis (Ol, 02) of 
the photographing optical lens system of the digital camera 
10 . 

Note, in Fig. 1, references WL indicate white lines 
15 painted along the sides of the roadway, and references WM 
indicate white traffic symbols painted on the roadway. 

Figures 2 and 3 show scenes of the pictures displayed 
on a monitor connected to the photogrammetric image processing 
computer system, which are indicated by references IM1 and IM2 . 
20 Of course, the respective scenes of the pictures IJal and IM2 , 
shown in Figs. 2 and 3, correspond to scenes photographed by 
the camera 10 at the photographing positions Ml and M2 . As 
shown in Fig. 2, the displayed picture IM1 is composed of JxK 
pixels , and a two-dimensional X a -Y a coordinate system is defined 
25 on the displayed picture IM1 as a picture coordinate system, 
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with the origin thereof being at a center of the picture IMl . 
Note, similarly for the picture IM2 shown in Fig. 3. 

Figure 4 perspectively shows the target 20 photographed 
in both the pictures IMl and IM2 . As is apparent from this 
5 drawing, the target 20 is formed as an L- shaped member including 
two bar elements 2 OA and 2 OB, which are joined to each other 
at their ends so as to form a right angle. The L-shaped member 
is reinforced with a strut element 20C bridging the bar elements 
20A and 20B. In this embodiment, the L-shaped member or target 

10 20 is covered with a suitable non-reflective black sheet. 
Alternatively, the L-shaped member may be painted black. 

The target 20 is provided with three main circular 
plate-like elements 22, 24 and 26 securely attached thereto, 
and respective three main reference points 22P, 24P and 26P 

15 are defined by the main circular plate-like elements 22, 24 
and 26. In this embodiment, each of the circular plate-like 
elements 22, 24 and 26 is covered with a suitable non-reflective 
black sheet, and each of the main reference points 22P, 2 4P 
and 26P is represented as a small white circle which is formed 

20 at the center of the corresponding circular plate-like element 
(22, 24, 26) . For example, the small white circle is defined 
by adhering a small circular reflective white sheet to the 
corresponding circular plate-like element (22, 24, 26) at the 
center thereof . Of course, alternatively, each of the circular 

25 plate-like elements 22, 24 and 26 may be painted black, and 
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each small white circle may be formed as a small white circle 
area painted at the center of the corresponding circular 
plate-like element (22, 24, 26). 

Note, in Fig. 2, the respective reference points 22P, 
5 2 4P and 2 6P of the target 2 0 are represented by coordinates 
a x (x al , y al ) , a 2 (x a2 , y a2 ) and a 3 (x a3 , y a3 ) based on the picture 
coordinate system (X a -Y a ) . 

A distance between the reference points 22P and 2 4P is 
equal to that between the reference points 24Pand 26P. Namely, 
10 the reference points 22P, 24P and 2 6P define apexes of an 
isosceles triangle. The distance between the reference points 
22P and 24P or between the reference points 24P and 2 6P is 
utilized as a standard measurement scale. 

Note, the reference points 22P, 24P and 26P are utilized 
15 to determine camera parameters necessary for production of a 
survey map based on one of the pictures IMl and IM2 , as explained 
in detail hereinafter. 

The target 2 0 is also provided with three assistant 
circular plate-like elements 32, 34 and 36 securely attached 
20 thereto, and respective three assistant reference points 32P, 
34P and 36P are defined by the assistant circular plate-like 
elements 32, 34 and 36. Each of the assistant circular 
plate-like elements 32, 34 and 36 is formed in substantially 
the same manner as the main circular plate-like element (22, 
25 24, 26) . As is apparent from Fig. 4, the assistant plate-like 
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elements 32 and 34 are arranged along the bar element 20A at 
regular intervals between the main circular plate-like element 
22 and 24, and the remaining assistant plate-like 36 is arranged 
on the bar element 2 OB at a center between the main circular 
5 plate-like elements 24 and 26. 

Note, the assistant reference points 32P, 34P and 36P 
are utilized in conjunction with the main reference points 22P, 
2 4P and 2 6P to confirm whether the determination of the camera 
parameters is correct, as explained in detail hereinafter. 

10 The target 20 is further provided with three protrusions 

2 0D protruded from the underside of the L-shaped member at 
respective locations corresponding to the main reference 
points 22P, 24Pand26P. The target 2 0 is located on the roadway 
such that the protrusions 2 0D are in contact with a surface 

15 of the roadway, resulting in a stable placement of the target 
2 0 on the roadway. 

The target 20 includes a tilt-angle sensor, as disclosed 
in Unexamined Japanese Patent Publication No. 10-185563. The 
tilt-angle sensor detects a tilt-angle of the target 20 to a 

20 horizontal plane when being located on the roadway, and outputs 
a tilt-angle signal representing the detected tilt-angle of 
the target 20. The target 20 also includes a signal processor 
for processing the signal output from the tilt-angle sensor, 
and a radio transmitter for transmitting the processed signal 

25 to a receiver provided in the camera 10 . The transmission of 
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the signal from the radio transmitter is repeated at regular 
intervals of very short time. Whenever a photographing 
operation is performed by the camera 10, the transmitted signal 
is retrieved from the receiver by the control circuit of the 
5 camera 10. The retrieved signal is processed by the control 
circuit of the camera 10, and is then stored in the memory card, 
together with a corresponding frame of image data obtained 
during the photographing operation. 

With reference Fig. 5, the photogrammetric image 

10 processing computer system according to the present invention 
is shown as a block diagram. 

As shown in Fig. 5, the image processing computer system 
comprises: a central processing unit (CPU) 40/ a display device 
42, such as a monitor; a display controller 44 for controlling 

15 the monitor 42; a display memory 46 for storing image data on 
which a scene to be displayed on the display device or monitor 
42 is based; an input device 48 including a keyboard and a mouse; 
an input-device controller 50 for controlling the input device 
48; a memory-card driver 52 for loading the memory card, 

20 indicated by reference 54; and a working memory 56 used as a 
cache memory when executing calculations and processings in 
the CPU 40 . The input-device controller 50, the working memory 
56, the display memory 46, the memory-card driver 52 and the 
display controller 4 4 are connected to the CPU 40 via a bus 

25 58 . 



As conceptually shown in Fig. 5, the CPU 40 includes 
an input-state managing section 40A, a display-state managing 
section 40B, a calculation-control section 40C and a data- 
managing section 40D. 
5 The input-state managing section 40A manages various 

information data and command data input through the input device 
48. For example, when a cursor or pointer is moved on a scene 
display on the monitor 42 by manipulating the mouse of the input 
device 48, the movement of the pointer is managed by the 

10 input-state managing section 40A. The input-state managing 
section 40A also monitors whether character code data is input 
via the keyboard of the input device 48 , and when character 
code data is input, the character code data is converted into 
character image data, which is output to the display memory 

15 46, whereby character images are displayed on the monitor 42. 

The display-state managing section 40B manages a scene 
to be displayed on the monitor 42. For example, when at least 
a part of the scene displayed on the monitor 42 is changed, 
the change of the scene is managed by the display-state managing 

20 section 40B. Namely, writing of image data in the display 
memory 4 6 is controlled by the display-state managing section 
40B, thereby changing the scene displayed on the monitor 42. 
The display controller 4 4 reads the image data from the display 
memory 46 at regular intervals of given short time, and converts 

25 the read image data into three primary (red, green and blue) 
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video data . 

The calculation-control section 40C executes various 
calculations and processings for producing a survey map of the 
roadway, as stated in detail hereinafter. 
5 The data-managing section 40D manages various data read 

from the memory card 54 through the memory-card driver 52, 
survey map data, various data necessary for producing the survey 
map data and so on . 

For the production of the survey map based on the pair 
10 of pictures IMl and IM2 (Figs. 2 and 3), initially, the 
photographing positions Ml and M2 and the photographing 
directions (which are represented by the orientations of the 
optical axes Ol and 02 , respectively) are spatially determined 
on the pictures IMl and IM2 . 
15 With reference to Fig. 6, the spatial determination of 

the photographing position Ml and the photographing direction 
Ol will be representatively explained below. 

Note, in this drawing, a positional relationship 
between the camera 10 and the target 2 0 is geometrically and 
20 conceptually shown when performing the photographing operation 
at the photographing position Ml . 

In Fig. 6, reference S indicates an image plane defined 
by the photographing optical lens system of the camera 10, 
reference C indicates a center of the image plane S, and 
25 reference f indicates a focal distance defined as a distance 
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between the center C of the image plane S and the photographing 
position (or back principal point) Ml. Of course, when the 
photographing operation is performed at the photographing 
position Ml, a photographed scene, formed on the image plane 
5 S, corresponds to the scene of picture IM1 displayed on the 
monitor 42 . 

As shown in Fig. 6, a three-dimensional X s -Y s -Z s 
coordinate system is defined on the target 2 0 as a target 
coordinate system such that the origin of the target coordinate 

10 system (X s -Y s -Z s ) is at the reference point 24P of the target 
20, with the X s -axis and the Z s -axis coinciding with a line 
segment between the reference points 22P and 2 4P and a line 
segment between the reference points 2 4P and 2 6P, respectively. 
Of course, both the X s -axis and the Z s -axis are perpendicular 

15 to the Y s -axis. 

The survey map is produced by projecting three- 
dimensional coordinates, representing an image point, on a 
plane defined by the X s -axis and the Z s -axis of the target 
coordinate system (X s -Y s -Z s ) , and the plane concerned must be 

20 horizontal before the production of the survey map can be 
properly performed. Nevertheless, the target 20 may not 
necessarily be horizontally positioned on the roadway. If the 
target 2 0 is tilted with respect to a horizontal plane, the 
plane concerned is also tilted. The tilt of the target 2 0 is 

25 corrected based on the tilt-angle data, which is derived from 
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the tilt-angle signal transmitted from the radio transmitter 
of the target 20. Thus, although the target 20 is tilted, it 
is possible to properly perform the definition of the target 
coordinate system (X s -Y s -Z s ) . 
5 As shown in Fig. 6, the distance between the reference 

points 22P and 24P is represented by reference LT , and thus 
the distance between the reference points 2 4P and 2 6P can be 
represented by the same reference LT , because the respective 
line segments between the reference points 22P and 24P and 

10 between the reference points 2 4P and 26P have the same length, 
as mentioned above. Accordingly, with respect to the target 
coordinate system <X S -Y S -Z S ) , the respective main reference 
points 22P, 24P and 2 6P are represented by the coordinates 
P sl (-LT, 0, 0) , P s2 (0, 0, 0) and P s3 (0, 0, LT) , as shown in Fig. 

15 6. Note, although not shown in Fig . 6, the respective assistant 
reference points 32P, 34P and 36P are represented by the 
coordinates P s4 (-2LT/3, 0, 0), P s5 (-LT/3, 0, 0) and P s6 (0 ( 0, 
LT/2) . 

In order to determine the photographing position (or 
20 back principal point) Ml with respect to the target coordinate 
system (X s -Y s -Z s ) , a two-dimensional X p -Y p coordinate system is 
defined on the image plane S as an image-plane coordinate system, 
and a three-dimensional X c -Y c -Z c coordinate system is defined 
on the camera 10 as a camera coordinate system, as shown in 
25 Fig. 6. 
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The origin of the image-plane coordinate system (X p -Y p ) 
is at the center C of the image plane S . When the photographing 
operation is performed at the photographing position Ml, the 
reference points 22P, 24P and 26P of the target 20, formed and 
5 recorded on the image-plane S, are represented by coordinates 
Px(x pl , y pl > , P 2 (x p2 , y p2 ) andp 3 (x p3 , y p3 ) based on the image-plane 
coordinate system (X p -Y p ) . The respective coordinates Pi(x pl , 
Y P i) / P 2 (x p2 , Y P 2> and p 3 (x p3 , y p3 ) essentially coincide with the 
coordinates a x (x al , y al ) , a 2 (x a2 , y a2 ) and a 3 (x a3 , y a3 ) based on the 

10 picture coordinate system (X a -Y a ) defined on the picture IMl 
displayed on the monitor 42 (Fig. 2) , because the image-plane 
coordinate system (X p -Y p ) and the picture coordinate system 
(X a -Y a ) are essentially identical to each other. 

As is apparent from Fig. 6, the origin of the camera 

15 coordinate system (X c -Y c -Z c ) coincides with the back principal 
point (Ml) , the Z c -axis coincides with the optical axis Ol , and 
the X c -axis and the Y c -axis are in parallel to the X p -axis and 
the Y p -axis of the two-dimensional coordinate system (X p -Y p ) , 
respectively. The origin of the camera coordinate system 

20 (X c -Y c -Z c ) or photographing position Ml, is represented by 

three-dimensional coordinates Ml (AX, AY, AZ) which are based 
on the target coordinate system (X s -Y s -Z s ) . An orientation of 
the Z c -axis or optical axis Ol is represented by three- 
dimensional angular coordinates (a, P, y) which are defined with 

25 respect to the target coordinate system (X s -Y s -Z s ) . Namely, the 
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Z c -axis or optical axis Ol defines angles of a, p and y with 
the X s -axis, Y g -axis and Z s -axis of the target coordinate system 
(X s -Y s -Z s ) , respectively. 

Note, the three-dimensional coordinates Ml (AX, AY, AZ) 
5 and the three-dimensional angular coordinates (a, p, y) are 
referred to as camera parameters (AX, AY, AZ , a, (3, y) for 
representing the photographing position Ml . 

When two-dimensional coordinates p 1 (x pi , y pi ) (i = 1, 2, 
3), representing the coordinates P!<x pl , y pl ) , p 2 (x p2 , y p2 ) and 
10 p 3 (x p3 , y p3 ) based on the image-plane coordinate system (X p -Y p ) , 
are expressed by three-dimensional coordinates P 01 (P cxl , p cyi r 
P czx ) based on the camera coordinate system (X c -Y c -Z c ) , the 
X-coordinate x pi and the Y-coordinate y pi are represented by the 
following formulas (1) and (2) , respectively 

X . = fx^- (1) 

15 P l J p [ 7 

czi 

y pi =f^ (2) 

Herein: "i" = 1, 2, 3, and "f" is the focal distance. 

On the other hand, the three-dimensional coordinates 
p =i( p cxi/ P c Y if P c Z1 ) based on the camera coordinate system 
20 (X c -Y c -Z c ) is expressed by three-dimensional coordinates P Si (P SX1 , 
p syi / p szi> based on the target coordinate system (X s -Y s -Z s ) , as 
follows : 
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R = 



P ci =R(P si -A) (3) 

' cosficosy cosasiny + sinasinficosy sinasiny - cosasinficosy "\ 

- cosjSsiny - cosacosy - sinasinfisiny sinacosy + cosasin/Ssiny 
sinp -s in acos ft cos acos/3 



A = 



AX 
AY 
AZ 



Herein: "R" is a rotational matrix of the angle a, p and y, and 
5 "A" is a vector representing a movement distance of the origin 
of the camera coordinate system (X c -Y c -Z c ) from the origin of 
the target coordinate system (X s -Y s -Z s ) . 

Thus, it is possible to calculate the three-dimensional 
coordinates Ml (AX, AY, AZ) and the three-dimensional angular 
10 coordinates (a, p , y) by the formulas (1) , (2) and (3) , whereby 
the photographing position Ml is spatially determined with 
respect to the target coordinate system (X s -Y s -Z s ) . 

In particular, the camera parameters (AX, AY, AZ , a, 
P, y) are calculated, using a sequential-approximation method. 
15 Namely, the calculation of the camera parameters (AX, AY, AZ , 
a, P, y) are performed such that a value O of the following 
formula (4) is minimized: 



®=ihi- x J + (y Pl -yJ} ( 4 > 



Note, at the beginning of the calculation, settings of 



suitable initial values are given to the camera parameters (AX, 
AY, AZ , a, P, y) . For example, in order to facilitate the 
sequential-approximation calculation, the camera parameters 
(AX, AY, AZ, a, p, y) maybe set as AX=Om, AY=1.5m, AZ=Om, a=0°, 

5 P=0°... 360° and y=0° . Namely, the initial value AY of 1.5m is 
selected as an approximate height of the camera 10 measured 
from the ground when photographing, and the initial value of 
P may be empirically selected from the range from 0° to 360° 
on the basis of the scene of the picture IMl . 

10 Note, of course, camera parameters (AX, AY, AZ , a, P, 

y) , representing the photographing position M2 of the picture 
IM2 , can be calculated in substantially the same manner as 
mentioned above . 

The calculated camera parameters (AX, AY, AZ , a, p, y) 

15 concerning the photographing positions Ml and M2 are once stored 
in working memory 56, and then the survey map of the roadway 
is produced, using the camera parameters (AX, AY, AZ , a, P, 
y) , by displaying a map-production scene on the monitor 42 , as 
shown , by way of example , in Fig . 7 . 

20 In particular, as shown in Fig. 7, the map-production 

scene includes a pair-picture-display area IMA and a map- 
production-display area DRA positioned below the pair- 
picture-display area IMA. The pair of pictures IMl and IM2 is 
displayed on the pair-picture-display area IMA, and an 

25 unfinished survey map is displayed on the map-production- 
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display area DRA. Note, the map-production-display area DRA 
corresponds to the plane defined by the X s -axis and the Z s - 
axis of the target coordinate system (X s -Y s -Z s ) . 

For example, an object point OPl , shown on the survey 
5 map, corresponds to a photographed object point OPl' on the 
picture IM1 , and corresponds to a photographed object point 
OPl" on the picture IM2 . By indicating the object points OPl' 
and OPl" by clicking on them with the mouse, a position of the 
object point OPl is determined on the map-production-display 
10 area DRA, using the camera parameters (AX, AY, AZ ,. a, P, j) . 
Also, an object point OP2 , shown on the survey map, corresponds 
to a photographed object point OP2 ' on the picture IMl , and 
corresponds to a photographed object point OP2" on the picture 
IM2 . Similarly, by indicating the object points OP2' and OP2" 
15 by clicking on them with the mouse, a position of the object 
point OP2 is determined on the map-production-display area DRA, 
using the camera parameters (AX, AY, AZ , a, P, y) . 

Note, in reality, when the object point (OPl', OPl", 
OP2 ' , OP2") is indicated by clicking on it with the mouse, the 
20 indicated object point is displayed as a colored s^all dot on 
the picture (IMl, IM2). 

After the determination of the two object points OPl 
and OP2 , a line segment LI is drawn and displayed between the 
object points OPl and OP2 , as shown in Fig. 7. The drawing of 
25 the line segment Ll is performed by previously selecting and 



setting a straight-line-drawing mode in a drawing-menu window 
DRM displayed on the map-production-display area DRA. 

The drawing-menu window DRM has plural item-buttons, 
representatively indicated by reference MIB , which correspond 
5 to various line-drawing modes. Of course, one of the line- 
drawing modes is the aforesaid straight-line-drawing mode, and 
other modes may be a curved-line-drawing mode, a diameter- 
designated-circular-drawing mode, a radius-designated- 
circular-drawing mode, a polygon-drawing mode and so on. Of 
10 course, one of the line-drawing modes is selected and set by 
operating a corresponding item button MIB by clicking on it 
with the mouse. Further, the drawing-menu window DRM has an 
input-fixing button IFB, which is operated by clicking on it 
with the mouse after object points necessary for drawing a line 
15 segment or line segments are displayed on the display area DRA. 

In short, in the aforesaid example, after the object 
points OP1 and OP2 are displayed on the map-production-display 
area DRA, the drawing of the line- segment Ll between the 
displayed object points OPl and OP2 is performed by the 
20 operation of the input-fixing button IFB provided that the 
straight-line-drawing mode is selected and set. Thus, by 
successively and correspondingly indicating plural pairs of 
object points, representing the roadway, on the pictures IMl 
and IM2 in the same manner as mentioned above, it is possible 
25 to draw and display various lines L on the map-production- 
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display area DRA. 

According to the present invention, as stated in detail 
hereinafter, the two-dimensional coordinates a x (x al/ y al ) , a 2 (x a2/ 
y a2 ) and a 3 (x a3 , y a3 ) are automatically and precisely read from 
5 the picture <IM1, IM2), and thus the camera parameters (AX, 
AY, AZ, a, p, y) can be accurately determined, whereby a reliable 
production of a survey map can be ensured. Also, since the 
accurate determination of the camera parameters (Ax, AY, AZ , 
a, P, y) are previously performed, resulting in a facilitation 
10 of the production of the survey map. 

Conversely, conventionally, the determination of the 
camera parameters <AX, AY, AZ , a, P, y) is performed by 
correspondingly indicating the reference points 22P, 24P and 
2 6P of the target 2 0 on the displayed pictures IMl and IM2 by 
15 clicking on them with the mouse. Namely, the determination of 
the camera parameters (AX, AY, AZ , a, P, y) is based on the 
two-dimensional coordinates a x (x al , y al ) , a 2 <x a2 , y a2 ) anda 3 (x a3 , 
y a3 ) obtained by the manual indication of the reference points 
22P, 24P and 26P of the target 20 on the displayed pictures 
20 IMl and IM2 in the same manner as the aforesaid manual indication 
of the object points (OPl' , OP2' ; OP2 ' , OP2") . Of course, 
unless the manual indication of the reference points 22P, 24P 
and 2 6P of the target 20 is precisely performed, the determined 
camera parameters are unreliable, due to the imprecise manual 
2 5 indication of the reference points , and thus a proper production 
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of the survey map cannot be ensured. 

Figure 8 shows a flowchart of a camera-parameter- 
determination routine, which forms a part of a main routine 
for producing a survey map based on a pair of pictures, and 
5 which is executed in the image-processing computer system shown 
in Fig. 5. 

At step S102, a frame of color image data is read from 
the memory card 54, and is stored in the working memory 56. 
Then, at step S104, a flag F is made to be "0". The flag F 

10 indicates whether a determination of camera-parameters (AX, 
AY, AZ, a, P, y) has been completed. Note, of course, when the 
determination of the camera parameters is incomplete, the 
setting of 0 is given to the flag F, and when the determination 
of the camera parameters has been completed, a setting of 1 

15 is given to the flag F, as explained hereinafter. 

At step S106, a frame of luminance image data is 
extracted from the frame of color image data, stored in the 
working memory 56, for example, corresponding to the picture 
IM1 photographed at the photographing position Ml, and is then 

20 stored in a JxK matrix-area defined in the working memory 56. 
In this embodiment, the frame of luminance image data is 
composed of the JxK pixels (Fig. 2) . Thus, the luminance image 
pixels included in one frame are stored in the JxK matrix- 
area of the working memory 56, and each image pixel with a 

25 certain luminance level is represented by a variable IM g (j, k) . 
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Note, in this embodiment, the luminance image pixels 
are sorted by 256 luminance levels, i.e. each image pixel is 
represented by 8 bits . 

At step S200, a circumscribed-rectangle-nomination 
5 subroutine is executed. By the execution of this subroutine, 
rectangles, each of which is presumed to be substantially 
circumscribed about an image of the target 20, are defined in 
and nominated from the JxK matrix-area of the working memory 
56. In short, the nomination of plural circumscr ibed- 
10 rectangles, each of which is presumed as including the image 
of the target 20, is performed. At this stage, it is impossible 
to find out which circumscribed-rectangle actually includes 
the image of the target 20. Note, the circumscribed- 
rectangle-determination subroutine is explained in detail 
15 hereinafter with reference to Fig. 9. 

At step S110 , it is determined whether all the nominated 
circumscribed-rectangles are processed, thereby finding out 
which circumscribed-rectangle encompasses the image of the 
target 20. Note, if there is no circumscribed-rectangle to be 
20 nominated by the execution of the circumscribed-rectangle- 
nomination subroutine, the camera-parameter-determination 
routine immediately ends, with the setting of "0" being given 
to the flag F, and an error message is displayed on the monitor 
42 to announce that the determination of the camera parameters 
25 is impossible. 
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At step S300, a threshold-nomination subroutine is 
executed, thereby nominating thresholds based on the image 
pixels, encompassed by one of the nominated circumscribed- 
rectangles , for performing binarization of that image pixels. 
Note, the threshold-nomination subroutine is explained in 
detail hereinafter with reference to Fig. 12. 

At step S120 , it is determined whether the image pixels , 
encompassed by the circumscribed-rectangle concerned, are 
processed with all the individual thresholds nominated by the 
execution of the thresholds-nomination subroutine (S300) . 

When it is confirmed that the image pixels are processed 
with all the individual thresholds, the control returns to step 
S110. Then, if the processings of all the nominated 
circumscribed-rectangles are not still completed, the 
threshold-nomination subroutine (S300) is further executed, 
thereby nominating some thresholds based on the image pixels 
encompassed by another one of the circumscribed-rectangles. 

At step 400, a provisional-target-image-determination 
subroutine is executed, thereby provisionally determining 
whether the circumscribed-rectangle concerned includes at 
least one image, which may be presumed as the image of the target 
20, based on the image pixels processed with one of the 
thresholds nominated by the execution of the thresholds- 
nomination subroutine (S300) . 

In the execution of the provisional-target-image- 
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determination subroutine (S400) , when the circumscribed- 
rectangle concerned includes an image, which may be presumed 
as the image of the target 20, the image is represented by six 
sets of coordinates, corresponding to the reference points 22P, 
5 24P, 26P, 32P, 34P and 36P of the target 2 0 , based on the picture 
coordinate system (X a -Y a ) . According to circumstances, two or 
more images, each of which may be presumed as the image of the 
target 20, are included in the circumscribed-rectangle 
concerned. In this case, the two or more groups of respective 
10 six sets of coordinates, corresponding to the reference points 
22P, 24P, 26P, 32P, 34P and 36P, are obtained by the execution 
of the provisional-target-image- determination subroutine. 

Note, the provisional-target-image-determination 
subroutine is explained in detail hereinafter with reference 
15 to Fig. 14. 

At step 130, it is determined whether all the groups 
of six sets of coordinates, corresponding to the reference 
points 22P, 24P, 26P, 32P, 34P and 36P, are processed, thereby 
confirming whether the respective six sets of coordinates, 
20 included in each group, truly represent the reference points 
22P, 24P, 26P, 32P, 34P and 36P. 

At step S130, when it is confirmed that all the groups 
of six sets of center-coordinates are processed, the control 
returns to step S120 . When the processing of the image pixels 
25 (encompassed by the circumscribed-rectangle concerned) with 
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all the individual thresholds is not still completed, the 
provisional-target-image determination subroutine (S400) is 
further executed, thereby provisionally determining whether 
the circumscribed-rectangle concerned includes at least one 
5 image, which may be presumed as the image of the target 20, 
using the image pixels processed with another one of the 
thresholds nominated by the execution of the thresholds- 
nomination subroutine (S300) . 

At step S500, a reference-point-coordinate-correction 
10 subroutine is executed, thereby correcting the six sets of 
coordinates, corresponding to the reference points 22P, 24P, 
26P, 32P, 34P and 36P of the target 20, obtained by the execution 
of the provisional-target-image-determination subroutine 
(S400) . Namely, it is possible to more accurately determine 
15 the six sets of coordinates by the execution of the 

reference-point-coordinate-correction subroutine . Note, the 
reference-point-coordinate-correction subroutine is 
explained in detail hereinafter with reference to Fig. 18. 
At step S600, a camera-parameter-calculation 
20 subroutine is executed, thereby calculating camera parameters 
(AX, AY, AZ, a, P, y) based on the three sets of coordinates, 
corresponding to the reference points 22P, 24P and 26P of the 
target 20. Note, the camera-parameter-calculation subroutine 
is explained in detail hereinafter with reference to Fig. 25. 
25 At step S700, a calculation-confirmation subroutine is 
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executed, thereby confirming whether the calculation of the 
camera parameters (AX, AY, AZ , a, P, y> is properly performed, 
using the three sets coordinates corresponding to the assistant 
reference points 32P, 34P and 36P of the target 20. Note, the 
5 calculation-confirmation subroutine is explained in detail 
hereinafter with reference to Fig. 26. 

At step S140, it is determined whether the calculation 
of the camera parameters of the (AX, AY , AZ , a , p , y) is correct. 
When the calculation of the camera parameters is correct, the 
10 control proceeds to step S142, in which the flag F is set to 
"1". On the other hand, if the calculation of the camera 
parameters is not correct, the control returns to step S110 
via steps S130 and S120, and the routine comprising steps S300, 
S120, S400, S130, S500, S600, S700 and S140 is executed with 
15 respect to another one of the circumscribed-rectangles 

nominated by the execution of the circumscribed-rectangle- 
nomination subroutine (S200) . 

Figure 9 shows a flowchart of the circumscribed- 
rectangle-nomination subroutine executed in step S2 0 0 of the 
20 camera-parameter-determination routine shown in Fig. 8. 

At step S202 , an 1/m reduced picture IG is produced from 
the JxK picture IMl , as conceptually shown in Fig. 10. Namely, 
one pixel is selected from m consecutive pixels in each of the 
horizontal lines of the picture IMl along the X a -axis of the 
25 picture coordinate system (X a -Y a ) , and one pixel is selected 
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from m consecutive pixels in each of the vertical lines of the 
picture IM1 along the Y a -axis of the picture coordinate system 
(X a -Y a ) . The selected pixels are stored in a sxt matrix area 
defined in the working memory 56, and each image pixel with 
5 a certain luminance level is represented by a variable IG 3 (s, 
t) . Note, of course, the reduced picture IG is composed of 
J/mxK/m pixels, and each element of the sxt matrix area is 
represented by 8-bits. 

Note, as shown in Fig. 10, a two-dimensional X as -Y as 
10 coordinate system is defined with respect to the reduced picture 
IG, an origin of which is at a center of the reduced picture 
IG. 

At step S204, a histogram is produced based on the 
reduced picture IG, using the variable IG s (s, t) , as shown, by 

15 way of example, in Fig. 11. In this histogram, the abscissa 
represents a distribution of luminance levels of all of the 
image pixels included in the reduced picture IG, and the 
ordinate represents a frequency or number of image pixels 
exhibiting a same luminance level. As stated above, the image 

20 pixels are sorted by the 256 luminance levels, which defines 
a full histogram-definition range from a zero luminance level 
to a 255 luminance level. Of course, as shown in Fig. 11, the 
minimum level (0) corresponds to black, and the maximum level 
(255) corresponds to white. 

25 At step S206, a maximum peak Pl^ of the histogram is 
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sought by suitably processing the histogram. Then, at step 
S208, a root luminance level of the maximum peak PK : , which is 
at a low luminance side thereof, is set as a threshold TH t . 
Namely, the threshold TH X is selected as a luminance level at 
5 which a frequency or number of pixels is abruptly increased 
in a band including the maximum peak PK X . 

At step S210, the image pixels included in the reduced 
picture IG are subjected to binarization with the threshold 
TH-l . The binarization of the image pixels with the threshold 
10 TH-l is performed, using the variable IG s (s, t) . For the 

binarization of the image pixels with the threshold TH X , a s' xt' 
matrix area, corresponding to the aforesaid sxt matrix area, 
is defined in the working memory 56. Each element of the s' xt' 
matrix area is represented by one bit, and is represented by 
15 a variable BIN (s' , t' ) . In the binarization, when each of the 
image pixels, represented by the variable IG s (s, t) , exhibits 
a luminance level more than the threshold TH X , a setting of u l" 
is given to a corresponding bit-element (pixel) represented 
by the variable BIN (s' , t' ) . On the other hand, when each of 
20 the image pixels, represented by the variable IG s (s, t) , 

exhibits a luminance level equal to or less than the threshold 
TH-l , a setting of "0" is given to a corresponding bit-element 
(pixel) represented by the variable BIN (s' , t' ) . 

At step S212, in the s'xt' matrix area, the bit-elements 
25 (pixels) having "0" are subjected to labeling. Namely, a 0-bit 
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area, in which bit-elements having "0" are consecutively and 
continuously arranged, is extracted from the s' xt' matrix area. 
Of course, usually, plural 0-bit areas are extracted from the 
s'xt' matrix area, and there may be a 0-bit area, corresponding 

5 to the image (black) of the target 20, among the extracted 0-bit 
areas. Also, in the labeling, a number of bits included in each 
0-bit area is counted. 

At step S214, small 0-bit areas, each of which is 
composed of a number of bits less than a predetermined number, 

10 i.e. each of which does not correspond to the image of the target 
20, is eliminated from among the extracted 0-bit areas. Of 
course, at this time, very small 0-bit areas (or noise areas) , 
each of which is composed of either one bit or several bits, 
are eliminated from among the extracted 0-bit areas. The 

15 remaining 0-bit areas are nominated as 0-bit areas, each of 
which may correspond to the image of the target image 20. 

At step S216, it is determined whether all the nominated 
0-bit areas are processed at steps S218 , S220 and S222 . Namely, 
the processings of all the nominated 0-bit areas are 

20 successively performed. Of course , when the processings of all 
the nominated 0-bit areas are completed, the control returns 
to the camera-parameter-determination routine . 

At step S218, a maximum X as -coordinate , a maximum 
Y as -coordinate , a minimum X as -coordinate and a minimum Y as - 

25 coordinate are read from each of the nominated 0-bit areas based 
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on the two-dimensional coordinate system (X as -Y as ) , which is 
defined with respect to the reduced picture IG . 

For example, when one of the nominated 0-bit areas 
corresponds to the image of the target 20, which is illustrated 
as a black-solid image in the reduced picture IG shown in Fig. 
10, a maximum X as - coordinate x asmax , a maximum Y as -coordinate y asmax , 
a minimum X as - coordinate x asmin and a minimum Y as -coordinate y asmin 
are read from the 0-bit area or black-solid image. Of course, 
a rectangle, which is defined by both the maximum coordinates 

( X asma X / Yasmax) and the minimUm Coordinates { X asmin , y asmin ) , is 

circumscribed about the black-solid image corresponding to the 
image of the target 20. 

At step S220, the respective maximum X as -coordinate , 
maximum Y as - coordinate , minimum X as -coordinate , and minimum 
Y as -coordinate , based on the two-dimensional coordinate system 
(X as -Y as ) , are converted into a maximum X a -coordinate , a maximum 
Y a -coordinate , a minimum X a -coordinate and a minimum Y a - 
coordinate based on the picture coordinate system (X a -Y a ) of 
the picture IMl . Namely, for example, the maximum X as - 
coordinate x asmax , maximum Y as -coordinate y asmax , minimum X as - 
coordinate x asmin , and minimum Y as -coordinate y asmin/ derived from 
the black-solid image corresponding to the image of the target 
20, are multiplied by the constant m as follows: 



x amin <- mxx asBin 
y Mln <- mxy asmin 
Thus, the respective maximum coordinates (x asmax , y asmax ) 
and minimum coordinates (x asmin , y asmin ) > based on the two- 
5 dimensional coordinate system (X as -Y as ) of the reduced picture 
IG, are converted into the maximum coordinates (x^^, y amax ) and 
minimum coordinates <x amin , y amin ) , based on the picture 
coordinate system (X a -Y a ) of the picture IM1 . 

Of course, as shown in Fig. 10, a rectangle GK is defined 
10 on the picture IMl by the converted maximum coordinates (x amax , 
y amax ) and minimum coordinates (x mB/ y amin ) , and is substantially 
circumscribed about a target image illustrated as a back-solid 
image in the picture IMl of Fig. 10. 

At step S222, the converted maximum coordinates and 
15 converted minimum coordinates are stored or recorded in a given 
area of the working memory 56. Of course, when the execution 
of this subroutine is completed, the maximum coordinates (x amax , 
Yamax) and minimum coordinates <x affiin , y amin ) / associated with the 
target image, are stored or recorded in the given area of the 
20 working memory 56. 

Figure 12 shows a flowchart of the threshold-nomination 
subroutine executed in step S300 of the camera-parameter- 
determination routine shown in Fig. 8. 

At step S302, all image pixels, encompassed by one of 
25 the nominated circumscribed-rectangles , are extracted from the 
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JxK matrix-area of the working memory 56, using the variable 
IM g (j, k) . For example, when the image pixels, encompassed by 
the circumscribed-rectangle GK including the target image, are 
extracted, the extraction of the image pixels from the 
circumscribed-rectangle GK is performed based on the maximum 
coordinates (x amax , y amax ) and the minimum coordinates (x amin , y amin ) , 
which define the circumscribed-rectangle GK . 

At step S304, a histogram is produced basad on the 
extracted image pixels, as shown, by way of example, in Fig. 
13. Then, at step S306, peaks PK 21 , PK 22 , PK 23 and PK 24 of the 
histogram are sought by suitably processing the histogram. 

Note, similar to Fig. 11, in the histogram of Fig. 13, 
the abscissa represents a distribution of luminance levels of 
all of the image pixels included in the circumscribed-rectangle 
concerned, and the ordinate represents a frequency or number 
of image pixels exhibiting a same luminance level. Also, the 
image pixels are sorted by the 256 luminance levels, which 
defines a full histogram-definition range from a zero luminance 
level to a 255 luminance level. Further, as shown in Fig. 13, 
the minimum level (0) corresponds to black, and the maximum 
level (255) corresponds to white. 

At step S308, it is determined whether all the peaks 
PK 21 , PK 22 , PK 23 and PK 24 are processed at steps S310 and S312 . 
Namely, the processings of all the peaks PK 21 , PK 22 , PK 23 and PK 24 
are successively performed one by one, and it is confirmed 
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whether the processings of all the peaks PK 21 , PK 22 , PK 23 and PK 24 
has been completed. Of course, when the processings of all the 
peaks PK 21 , PK 22 , PK 23 and PK 24 are completed, the control returns 
to the camera-parameter-determination routine. 

Note, if it is determined that the histogram includes 
no peak (S306) , it is regarded that the processings of the peaks 
has been completed, whereby the control immediately returns 
from step S308 to the camera-parameter-determination routine. 

At step S310, a root luminance level of each peak (PK 21 , 
PK 22 , PK 23 , PK 24 ) , which is at a high luminance side thereof, is 
set as a threshold (TH 21 , TH 22 , TH 23 , TH 24 ) . Namely, the threshold 
(TH 21 , TH 22 , TH 23 , TH 24 ) is selected as a luminance level at which 
a variation of the frequency or number of pixels is small. 

At step S312, the set threshold (TH 21 , TH 22 , TH 23 , TH 24 ) 
is stored and recorded in a given area of the working memory 
56 . 

Figure 14 shows a flowchart of the provisional- 
target-image-determination subroutine executed in step S400 
of the camera-parameter-determination routine shown in Fig. 
8 . 

At step S402, the image pixels, encompassed by the 
circumscribed- rectangle concerned, are subjected to 
binarization with one of the nominated thresholds TH 21 , TH 22 , 
TH 23 and TH 24 . For the binarization, a matrix area, 
corresponding to the circumscribed-rectangle concerned, is 
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defined in the working memory 56, and each element of the matrix 
area is represented by one bit. In the binarization , when each 
of the image pixels exhibits a luminance level equal to or more 
than the threshold <TH 21 , TH 22 , TH 23 , TH 24 ) , a setting of "1" is 
5 given to a corresponding bit-element (pixel) of the matrix area . 
On the other hand, when each of the image pixels exhibits a 
luminance level less than the threshold (TH 21 , TH 22 , TH 23 , TH 24 ) , 
a setting of "0" is given to a corresponding bit-element (pixel) 
of the matrix area. 
10 Note, for example, the binarization of the image pixels , 

encompassed by the circumscribed-rectangle GK (Fig. 10) is 
performed, using the variable IM g (j, k) , based on the maximum 
coordinates (x amax , y amax ) and the minimum coordinates (x amin , y amin ) 
which define the circumscribed-rectangle GK . 
15 At step S404, in the matrix area, the bit-elements 

(pixels) having "1" is subjected to a labeling. Namely, an 
1-bit area, in which bit-elements having "1" are consecutively 
and continuously arranged, is extracted from the matrix area. 
Note, of course, when the bit-elements of the matrix area, 
20 corresponding to the circumscribed-rectangle GK including the 
target image, are subjected to the labeling, at least six 1-bit 
areas, representing the reference points 22P, 24P, 26P, 32P, 
34P and 36P of the target 20, may be extracted from the matrix 
area . 

25 At step S40 6, respective groups of image pixels, 
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corresponding to the extracted 1-bit areas, are extracted from 
the circumscribed-rectangle concerned, and are then stored and 
recorded in a given area of the working memory 56. 

At step S4100, a provisional-balance-center- 
determination subroutine is executed, whereby a center of 
balance of each of the extracted 1-bit areas is provisionally 
determined as a set of coordinates based on the picture 
coordinate system (X a -Y a ) , by suitably processing a 
corresponding group of image pixels, each of which is 
represented by the variable IM g (j, k) . Of course, when the 
groups of image pixels are included in the circumscribed- 
rectangle GK, the set of determined coordinates may represent 
one of the reference points 22P, 24P, 26P, 32P, 34P and 36P 
of the target 20. Note, the provisional-balance-center- 
determination subroutine is explained in detail hereinafter 
with reference to Fig. 15. 

At step S4200, a balance-center-alignment-detection 
subroutine is executed, whereby an alignment of three 
balance-centers and an alignment of four balance-centers are 
detected from among the balance-centers included in the 
circumscribed-rectangle concerned. Of course, when the 
balance-centers , determined by the execution of the 
provisional-balance-center-determination subroutine, are 
included in the circumscribed-rectangle GK, the alignment of 
three balance-centers may correspond to one of three alignments 
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of the reference points 22P, 32P and 34P; 32P, 34P and 24P; 
and 24P, 36P and 2 6P, and the alignment of four balance-centers 
may correspond to an alignment of the reference points 22P, 
32P, 34P and 24P (Fig. 4). Note, the balance-center- 
alignment-detection subroutine is explained in detail 
hereinafter with reference to Fig. 16. 

At step S4300, a provisional-target-image- 
confirmation subroutine is executed to provisionally confirm 
whether the image of the target 20 is included in the 
circumscribed-rectangle concerned, by suitably processing the 
alignments of three balance-centers and the alignments of four 
balance-centers obtained by the execution of the balance- 
center-alignment-detection subroutine . 

Figure 15 shows a flowchart of the provisional- 
balance-center-determination subroutine executed in step 
S4100 of the provisional-target-image-determination 
subroutine of Fig. 14. 

At step S4102, it is determined whether all the groups 
of image pixels, corresponding to the extracted 1-bit areas, 
are processed for the provisional determination of a center 
of balance of each of the extracted 1-bit areas. Of course, 
when the processings of all the groups of image pixels are 
completed, the control returns from step S4102 to the 
target-image-determination subroutine of Fig. 14. Note, if 
there is no 1-bit area to be executed from the aforesaid matrix 
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area (step S404 of Fig. 14) , i.e. if there is no group of image 
pixels to be processed, the control immediately returns from 
step S4102 to the provisional- target-image-determination 
subroutine of Fig. 14. 
5 At step S4104, each of variables SumX, SumY and SumIM g 

are initialized to "0". Then, at step S4106, it is determined 
whether all the image pixels included in the group concerned 
are processed at step S4108 . 

At step S4108, the following calculations are executed 
10 with respect to all the image pixels included in the group 
concerned : 

SumX <r- SumX + IM g {j, k) x j 
SumY <— SumY + IM g (j, k) xk 
SumIM g <- SumIM g + IM g (j, k) 
15 Namely, a luminance level of each image pixel is 

multiplied by an X a -coordinate "j" thereof, and the variable 
SumX is ultimately defined as a total of the multiplied 
luminance levels. Similarly, a luminance level of each image 
pixel is multiplied by a Y a -coordinate "k" thereof, and the 
20 variable SumY is ultimately defined as a total of the multiplied 
luminance levels. Also, the variable SumIM g is ultimately 
defined as a total of luminance levels of all the image pixel 
included in the group concerned. 

At step S4106, when it is confirmed that the execution 
25 of the aforesaid calculations is completed with respect to all 
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the image pixels, included in the group concerned, the control 
proceeds from step S4106 to step S4110, in which a set of 
coordinates (x an/ y an ) , provisionally representing the center 
of balance of the extracted 1-bit area corresponding to the 
group concerned, is calculated as follows: 
x an <r- SumX/SumIM g 
y an <- SumY/ SumIM g 
The set of calculated center-coordinates (x an , y an ) is 
stored and recorded in a given area of the working memory 56. 
Of course, when a number of the extracted 1-bit areas or groups 
of image pixels is N, N sets of center-coordinates (x an , y an ) 
(n = 1, 2, ... N) can be obtained by the execution of the 
provisional-balance-center-determination subroutine . 

Figure 16 shows a flowchart of the balance-center- 
alignment-detection subroutine executed in step S4200 of the 
provisional-target-image-determination subroutine of Fig. 14. 

At step S4201, all combinations of two sets of 
balance-center-coordinates are produced from among the N sets 
of balance-center-coordinates (x an , y an ) . Then, at step S4202 , 
it is determined whether all the combinations of two sets of 
balance-center-coordinates are processed at steps S4204 , S4206 , 
S4208, S4210, S4212 and step S4214. When the processings of 
all combinations of two sets of balance-center-coordinates are 
completed, the control returns from step S42 02 to the 
provisional-target-image-determination subroutine of Fig. 14 . 



At step S4204, a line-segment is produced between two 
sets of balance-center-coordinates . Then, at step S4206, 
another set of balance-center coordinates, generally aligned 
along the line-segment between the aforesaid two sets of 
5 balance-center-coordinates, is detected. In other words , the 
detection of the set of balance-center coordinates is performed 
with a predetermined tolerance . 

At step S4208, it is determined whether a number of sets 
of balance-center-coordinates, generally aligned with each 
10 other along the line-segment is, three or not. If the number 
is three, the control proceeds to step S4210, in which the three 
sets of balance-center-coordinates are stored and recorded as 
an alignment of three balance-centers in a given area of the 
working memory 56. 
15 On the other hand, at step S4208, if the number is not 

three, the control skips step S4212, in which it is determined 
whether the number of sets of balance-center-coordinates, 
aligned with each other along the line-segment, is four or not. 
If the number is four, the control proceeds to step S4214, in 
20 which the four sets of balance-center-coordinates are stored 
and recorded as an alignment of four balance-centers in a given 
area of the working memory 56. 

Thereafter, the control returns to step S4202 . Also, 
at step S4212, if the number is not four, the control returns 
25 to step S4202. Namely, the routine comprising steps S4202, 
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S4206, S4208, S4210, S4212 and S4214 is repeated until the 
processings of all combinations of two sets of balance- 
center-coordinates are completed. 

Note, when all the sets of balance-center-coordinates 
5 are included in the circumscribed-rectangle GK (Fig. 10), at 
least three alignment of three balance-centers, corresponding 
to respective alignments of the reference points 22P, 32P and 
34P; 32P, 34P and 24P; and 24P, 36P and 26P, may be obtained, 
and at least one alignment of four balance-centers, 

10 corresponding to an alignment of the reference points 22P, 32P, 
34P and 24P, may be obtained. 

Figure 17 shows a flowchart of the provisional- 
target-image-confirmation subroutine executed in step S4300 
of the provisional-target-image-determination subroutine of 

15 Fig. 14. 

At step S4302, it is determined whether all the 
alignments of three balance-centers are processed. If the 
processings of all the alignments of three balance-centers is 
not completed, i.e. if an alignment of three balance-centers 

20 to be processed remains, the control proceeds to step S4303, 
in which the alignment of three balance-centers to be processed 
is selected. Note, if the processings of all the alignments 
of three balance-centers are completed, i.e. if there is no 
alignment of three balance-centers to be processed, the control 

25 returns from step 4302 to the provisional-target-image- 
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determination subroutine of Fig. 14. 

At step S4304, it is determined whether all the 
alignments of four balance-centers have been processed. If the 
processings of all the alignments of four balance-centers are 
5 not completed, i.e. if an alignment of four balance-centers 
to be processed remains, the control proceeds to step S4305, 
in which the alignment of four balance-centers to be processed 
is selected. Note, if the processings of all the alignments 
of four balance-centers are completed, i.e. if there is no 
10 alignment of four balance-centers to be processed, the control 
returns from step S4304 to step S4302 . 

At step S4306, it is determined whether an endmost 
balance-center of the selected alignment of three balance- 
centers and an endmost balance-center of the selected alignment 
15 of four balance-centers coincides with each other. If there 
is no coincidence, the control returns to step S4304 . 

At step S4306, if there is a coincidence between the 
endmost balance-center of the selected alignment of three 
balance-centers and the endmost balance-center of the selected 
20 alignment of four balance-centers, the control proceeds to step 
S4308, in which an angle 6 is measured from the alignment of 
four balance-centers to the alignment of three balance-centers 
along the counterclockwise direction. 

At step S4310, it is determined whether the measured 
25 angle 9 falls in a range between angles 0 L and 0 2 . The angle 
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Q 1 may be set to be 0°, and the angle 0 2 may be set to be 180°, 
because the right corner angle of the target 20 is photographed 
and recorded as an angle between 0° and 180°. 

If the measured angle 0 falls in the range between the 
angles Q 1 and 0 2 , it is provisionally confirmed that the 
circumscribed-rectangle concerned includes the image of the 
target 20. Thus, the control proceeds from step S4310 to step 
S4312, in which the respective six balance-centers, included 
in both the selected alignments, are stored and recorded in 
a given area of the working memory 5 6 as six sets of 
center-coordinates corresponding to the reference points 22P, 
24P, 26P, 32P, 34P and 36P based on the picture coordinate system 
(X a -Y a ) . On the other hand, if the measured angle 9 does not 
fall in the range between the angles 0! and 0 2 , the control 
returns to step S4304 . 

Of course, according to circumstances, there may be a 
case where the circumscribed-rectangle concerned may include 
two or more images , each of which may be presumed as the image 
of the target 20 . In this case , two or more groups of respective 
six sets of center-coordinates corresponding to the reference 
points 22P, 24P, 26P, 32P, 34P and 36P are provisionally 
determined by the execution of the provisional-target- 
image-confirmation subroutine of Fig. 17, and respective six 
sets of center-coordinates, included in each group, are 
represented by variables va x (x al , y al ) , va 2 (x a2 , y a2 ) , va 3 (x a3 , y a3 ) , 



48 



va 4 (x a4 , y a4 ) , va 5 (x a5f y a5 ) and va 6 (x a6/ y aS ) based on the picture 
coordinate system (X a -Y a ) . 

Figure 18 shows a flowchart of the reference-point- 
coordinate-correction subroutine executed in step S500 of the 
5 camera-parameter-determination routine shown in Fig. 8. 

At step 502, it is determined whether all the six groups 
of image pixels, from which the six sets of center-coordinates 
are derived, respectively, are processed. If the processings 
of the six groups of image pixels are completed, the control 
10 returns from step S502 to the camera-parameter-determination 
routine shown in Fig. 8. 

At step 504, an image-pixel area, encompassing one of 
the six groups of image pixel, is extracted from the picture 
IM1 . In this embodiment, the extracted image-pixel area may 
15 correspond to a 13x10 matrix area, as shown, by way of example, 
in Fig. 19, the image-pixel area being indicated by reference 
KR. An extent of the image-pixel area is suitably selected as 
an area which surrounds each of the reference points 22P, 24P, 
26P, 32P, 34P and 36P, but is smaller than a photographed image 
20 of a corresponding circular plate-like element (22, 24, 26, 
32, 34, 36) of the target 20. 

In the image-pixel area KR, each of the image pixels 
is shown as a small square. An open square represents an image 
pixel with a white level, and a hatched square represents an 
25 image pixel with a gray level. The narrower a hatching pitch 
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of the hatched square, the higher the gray level of the image 
pixel represented by the hatched square. Note, as already 
stated above, the image pixels are sorted by 256 luminance 
levels, and each image pixel is represented by the variable 
5 IM g (j, k) . 

At step S506, a histogram is produced based on the image 
pixels included in the extracted image-pixel area KR, using 
the variable IM g (j, k) , as shown, by way of example, in Fig. 
20. Of course, in this histogram, the abscissa represents a 

10 distribution of luminance levels of the image pixels included 
in the image-pixel area KR, and the ordinate represents a 
frequency or number of image pixels exhibiting a same luminance 
level. Also, a minimum level (0) corresponds to black, and a 
maximum level (255) corresponds to white. 

15 As is apparent from Fig. 20, the produced histogram 

features a low luminance level peak PK 31 and a high luminance 
level peak PK 32 , because the image-pixel area is selected as 
the area which surrounds each of the reference points 22P, 24P, 
2 6P, 32P, 34P and 36P, but is smaller than the photographed 

20 image of the corresponding circular plate-like element (22, 
24, 26, 32, 34, 36) of the target 20. 

At step S508, the low luminance level peak PK 31 is 
detected from the histogram as shown in Fig. 20. Then, at step 
S510, a luminance level, corresponding to the low luminance 

25 level peak PK 31 , is set as a threshold TH 3 . 
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At step S512, the image pixels, included in the 
image-pixel area KR, are subjected to the Laplacian-f iltering 
processing, using, for example, an 8-neighbor Laplacian filter 
as conceptually shown in Fig. 21. Namely, in the image-pixel 
5 area KR, each of the image pixels, represented by the variable 
IM g (j, k) , is converted into a variable KR ( j , k) , using the 
following formula: 

KR 0, k)= 9 x IM g (j, k ) 

+ (-;)* i IM g U- ] > k + i)+ im g {j, k + i) 

+ IM g (j + l k + l)+ IM g (j - 1, k) 
+ IM g (j+ 1, k)+ IM g (j -1, k-l) 
+ IM g (j, k-l)+ IM g (j + 1, k-l)] 

By the Laplacian-f iltering processing, the image of the 
10 image-pixel area KR is sharpened, as conceptually shown in a 
graph of Fig. 22. In this graph, there is shown a variation 
in the luminance level along a line, indicated by an arrow LX 
in Fig. 19. Namely, a solid curve represents the variation in 
the luminance level when the image pixels is not subjected to 
15 the Laplacian-f iltering processing, and a broken curve 

represents the variation in the luminance level after the image 
pixels are subjected to the Laplacian-f iltering processing. 
Of course, due to Laplacian-f iltering processing, not only can 
the image of the image-pixel area KR be sharpened, but noise 
20 can also be eliminated from the image-pixel area KR. 

At step S514, the processed image pixels of the 
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image-pixel area KR are subjected to binarization with the 
threshold TH 3 . For the binarization, a matrix area, 
corresponding to the image-pixel area KR, is defined in the 
working memory 56, and each element of the matrix area is 
5 represented by one bit. In the binarization, when each of the 
image pixels exhibits a luminance level equal to or more than 
the threshold TH 3 , a setting of "1" is given to the corresponding 
bit-element (pixel) of the matrix area. On the other hand, when 
each of the image pixels exhibits a luminance level less than 
10 the threshold TH 3 , a setting of "0" is given to the corresponding 
bit-element (pixel) of the matrix area. 

At step S516, in the matrix area, the bit-elements 
(pixels) having "1" is subjected to labeling. Namely, an 1-bit 
area, in which bit-elements having "1" are consecutively and 
15 continuously arranged, is extracted from the matrix area. 

Of course, when the bit-elements of the matrix area, 
corresponding to the image-pixel area KR, are subjected to the 
labeling, an 1-bit area, representing a corresponding 
reference point (22P, 24P, 26P, 32P, 34P, 36P) of the target 
20 20, is extracted from the matrix area. Nevertheless, another 
1-bit area may be extracted as noise from the matrix area. For 
example, when a whitish scrap has adhered to the corresponding 
circular plate-like element (22, 24, 26, 32, 34, 36) of the 
target 20, an area corresponding to the whitish scrap may be 
25 extracted as the 1-bit area from the matrix area. 
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At step S518, respective groups of image pixels, 
corresponding to the extracted 1-bit areas, are extracted from 
the image-pixel area, and are then stored or recorded in a given 
area of the working memory 56. 
5 At step S5100, a precise-balance-center-determination 

subroutine is executed, to thereby precisely determine a center 
of balance of each of the extracted 1-bit areas as a set of 
coordinates based on the picture coordinate system <X a -Y a ) by 
suitably processing a corresponding group of image pixels, each 
10 of which is represented by the variable IM g (j, k) . Note, the 
precise-balance-center-determination subroutine is explained 
in detail hereinafter with reference to Fig. 23. 

At step S5200, a balance-center-selection subroutine 
is executed, whereby a most precise set of center-coordinate 
15 is selected from among plural sets of coordinates obtained 
during the execution of the precise-balance-center- 
determination subroutine, each set of coordinates being 
presumed as representing a center of balance of one of the 
extracted 1-bit areas. Note, the balance-center-selection 
20 subroutine is explained in detail hereinafter with reference 
to Fig. 24. 

Figure 23 shows a flowchart of the precise-balance- 
center-determination subroutine executed in step S5100 of the 
reference-point-coordinate-correction subroutine of Fig. 18. 
25 At step S5102, it is determined whether all the groups 
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of image pixels, corresponding to the extracted 1-bit areas, 
are processed for the precise determination of a center of 
balance of each of the extracted 1-bit areas. Of course, when 
the processings of all the groups of image pixels are completed, 
5 the control returns from step S5102 to the reference- 
point-coordinate-correction subroutine of Fig. 18. Note, if 
there is no 1-bit area to be executed from the aforesaid matrix 
area (step S516 of Fig. 18) , i.e. if there is no group of image 
pixels to be processed, the control immediately returns from 

10 step S5102 to the reference-point-coordinate-correction 
subroutine of Fig. 18. 

At step S5104, each of variables SumX, SumY and SumIM g 
are initialized to "0". Then, at step S5106, it is determined 
whether all the image pixels included in the group concerned 

15 are processed at step S5108. 

At step S5108, the following calculations are executed 
with respect to all the image pixels included in the group 
concerned : 

SumX <r- SumX + [IM g (j, k) - TH 3 ] x j 
20 SumY <- SumY 4- [IM g (j, k) - TH 3 ] xk 

SumIM g <- SumIM g + [IM g (j, k) - TH 3 ] 
Namely, a subtraction of the threshold TH 3 from a 
luminance level of each image pixel is multiplied by an X a - 
coordinate "j" thereof, and the variable SumX is ultimately 
25 defined as a total of the multiplied luminance levels . 
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Similarly, a subtraction of the threshold TH 3 from a luminance 
level of each image pixel is multiplied by a Incoordinate "k" 
thereof, and the variable SumY is ultimately defined as a total 
of the multiplied luminance levels. Also, the variable SumIM g 
5 is ultimately defined as a total of luminance levels of all 
the image pixels, from each of which the threshold TH 3 is 
subtracted. 

At step S5106, when it is confirmed that the execution 
of the aforesaid calculations is completed with respect to all 
10 the image pixels, included in the group concerned, the control 
proceeds from step S5106 to step S5110 , in which a set of 
center-coordinates (x am , y am ) , representing the center of 
balance of the extracted 1-bit area corresponding to the group 
concerned, is calculated as follows: 
15 x am <- SumX/ SumIM g 

y am <— SumY/SumIM g 
The set of calculated center-coordinates (x am , y ara ) is 
stored and recorded in a given area of the working memory 56. 
Of course, when a number of the extracted 1-bit areas or groups 
20 of image pixels is M, M sets of center-coordinates (x am , y am ) 
(m = 1 , 2, ... M) can be obtained by the execution of the 
precise-balance-center-determination subroutine . 

Note, although the calculations of the center- 
coordinates (x am , y am ) is similar to the calculations of the 
25 provisional center-coordinates (x an , y an ) performed in the 
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provisional -balance-center-determination subroutine of Fig. 
15, it is possible to more precisely determine the center- 
coordinates (x ani , y am ) in comparison with the provisional 
center-coordinates (x an , y an ) , due to the subtraction of the 
5 threshold TH 3 from a luminance level of each image pixel (step 
S5108) . 

Figure 2 4 shows a flowchart of the balance-center- 
selection subroutine executed in step S5200 of the 
reference-point-coordinate-correction subroutine of Fig. 18. 

10 At step S5202, a variable "MinLength" is initialized 

to be a suitable large value . For example , the variable 
MinLength may be initially set to be a distance corresponding 
to 13 image pixels arranged along the X a -axis in the image- 
pixel area KR (Fig. 19) . 

15 At step S5204, it is determined whether all the M sets 

of center-coordinates (x am , y a J (m = 1 , 2, ... M) are processed 
at steps S5205, S5206, S5208 and S5210. When it is confirmed 
that the processings of all the M sets center coordinates (x am , 
y am ) are completed, the control returns from step S5204 to the 

20 reference-point-coordinate-correction subroutine of Fig. 18. 

At step S52 05 , a distance AL between a set of coordinates 
va x (x ai , y ai ) and one of the M sets of coordinates (x am/ y am ) is 
calculated. The set of coordinates va 1 (x ai/ y ai ) represents one 
of the six sets of center-coordinates va x (x al , y al ) , va 2 (x a2 , y a2 ) , 

25 va 3 (x a3 , y a3 ) , va 4 (x a4 , y a4 ) , va 2 (x a5/ y aS ) andva s (x aS , y a6 ) , 
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corresponding to the reference points 22P, 24P, 26P, 32P, 34P 
and 36P of the target 20, obtained by the execution of the 
provisional-target-image-confirmation subroutine of Fig. 17. 
For example, when the image-pixel area KR (Fig. 19) , from which 
5 the M sets of center-coordinates (x am , y am ) are derived, includes 
an image corresponding to the reference point 22P of the target 
20, the distance AL is calculated as a distance between the 
set of center-coordinates va x (x al , y al ) and one of the M sets 
of coordinates (x am , y am ) . 

10 At step S5206, it is determined whether the variable 

MinLength is larger than the distance AL . Initially, since the 
variable MinLength has the suitable large value, which may be 
set as the distance corresponding to 13 image pixels arranged 
along the X a -axis in the image-pixel area KR (MinLength > AL) , 

15 the control proceeds to step S5208, in which the variable 
MinLength is made to be AL . 

At step S5210, the following calculation is executed: 
va ± (x ai , y a J <- (x am , y am ) 
Thereafter, the control returns to step S5204, and the routines 

20 comprising steps S5204, S5205, S5206, S5208 and S5210 are 

repeatedly executed until the processings of all the M sets 
of center coordinates (x am , y am ) are completed. 

Thus, when the processings of all the M sets of center 
coordinates (x^, y am ) are completed, a most precise set of 

25 coordinates for representing a reference point (22P, 24P, 26P, 
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32P, 34P, 36P) of the target 20 is selected from among the M 
sets center coordinates (x am , y am ) , and a corresponding set of 
coordinates va x (x ai , y ai ) , representing one of the reference 
points 22P, 24P, 26P, 32P, 34P and 36P, is corrected to the 
5 most precise set of coordinates . 

Figure 2 5 shows a flowchart of the camera- 
parameter-calculation subroutine executed in step S600 of the 
camera-parameter-determination routine shown in Fig. 8. 

At step S602, the camera parameters (AX, AY, AZ , a, P, 
10 y) are initialized to "0". 

At step S604, three-dimensional coordinates P cx (P cxx , P cyx / 
P czi ) (i = 1, 2, 3) of the main reference points 22P, 24P and 
2 6P, based on the camera coordinate system (X c -Y c -Z c ) , are 
calculated by providing the aforesaid formula (3) with the 
15 coordinates P sl (-LT, 0, 0), P s2 <0, 0, 0) and P s3 (0, 0, LT) , and 
by giving respective settings of suitable values to the camera 
parameters AX, AY, AZ , a, P and y. 

At step S606, two-dimensional coordinates p^fx^, y pi ) 
(i = 1, 2, 3) of the main reference points 22P, 24P and 26P, 
20 based on the image-plane coordinate system (X p -Y p ) , are 

calculated by providing the aforesaid formulas (1) and (2) with 
the calculated three-dimensional coordinates P CX (P CX1 , P oyx , P CZ1 ) 
(i = 1, 2, 3) . 

At step S608, the following calculation is executed: 
25 d <- |va x - pj + |va 2 - p 2 | + |va 3 - p 3 | 
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Herein: of course, the respective p lr p 2 and p 3 are the 
coordinates (x pl , y pl ) , (x p2 , y p2 ) and (x p3/ y p3 ) of the main 
reference points 22P, 24P and 26P calculated at step S606 and 
based on the image-plane coordinate system (X p -Y p ) , and the 
5 respective va 1( va 2 and va 3 are the center-coordinates (x al/ y al ) , 
(x a2 , y a2 ) and (x a3 , y a3 ) of the main reference points 22P, 2 4P 
and 26P obtained from the picture IMl and based on the picture 
coordinate system (X 3 -Y a ) . Note, as already stated above, the 
image-plane coordinate system (X p -Y p ) and the picture 
10 coordinate system (X a -Y a ) are essentially identical to each 
other . 

At step S610, it is determined whether the calculated 
"d" is minimized. If the calculated "d" is not minimized, the 
control returns to step S604, and the routine comprising steps 

15 S604, S606 and S608 is further executed by giving respective 
settings of other values to the camera parameters AX, AY, AZ , 
a, P and y. Namely, the execution of the routine comprising 
steps S604 , S606 , S608 and S610 is repeated until the calculated 
"d" is minimized. 

20 At step S610, when the minimization of the calculated 

"d" is confirmed, the control proceeds from step S610 to step 
S612, in which the camera parameters AX, AY, AZ , a, P and y as 
they stand now are stored and recorded in a given area of the 
working memory 56. Thereafter, the control returns to the 

25 camera-parameter-determination routine of Pig. 8. 
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Figure 2 6 shows a flowchart of the calculation- 
confirmation subroutine executed in step S700 of the 
camera-parameter-determination routine shown in Fig . 8 . 

At step 702, it is determined whether each of the 
5 calculated camera parameters AX, AY, AZ , a, (3 and y is converged 
on a predetermined value. If a sufficient convergence of the 
camera parameters is not obtained, the control skips to step 
S714, in which a flag DF is set to "0". The flag DF indicates 
whether the calculation of the camera parameters AX, AY, AZ , 

10 a, 3 and y is correct. When the calculation of the camera 
parameters is not correct, the flag DF is set to "0", and when 
the calculation of the camera parameters is correct, the flag 
DF is set to "1". 

At step S702, when the sufficient convergence of the 

15 camera parameters is obtained, the control proceeds to step 
S704, in which it is determined whether all two-dimensional 
coordinates p 1 (x pi , y p;L ) (i = 4, 5, 6) of the assistant reference 
points 32P, 34P and 36P, based on the image-plane coordinate 
system (X p -Y p ) , have been calculated at steps S706 and S708. 

20 When the calculations of all the two-dimensional 

coordinates p x (x pi , y pi ) (i = 4, 5, 6) is not completed, the 
control proceeds to step S706, in which one of three-dimensional 
coordinates P C1 (P C3C1/ P cyi , P czi ) (i = 4 , 5, 6) of the assistant 
reference points 32P, 34P and 36P is calculated by providing 

25 the aforesaid formula (3) with a corresponding one set of the 
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three sets of coordinates P s4 (-2LT/3, 0, 0), P s5 (-LT/3 / 0, 0) 
and P s6 {0, 0, LT/2) and the camera parameters AX, AY, AZ , <X, 
fj and y obtained by the execution of the camera-parameter- 
calculation subroutine of Fig. 25. 
5 At step S708, a set of two-dimensional coordinates p x (x pi , 

y ) (i = 4, 5, 6), representing the corresponding assistant 
reference point (32P, 34P, 36P) , is calculated by providing 
the aforesaid formulas (1) and (2) with the calculated 
three-dimensional coordinates P ci (P CXi , P eyx , P CZ i) (i = 4 ' 5 / 6 > ■ 
10 At step S710, the following calculation is executed: 

Ae <— |va x - pj {i = 4, 5, 6) 
Herein: of course, "p ± " is the coordinates (x pi , y pi ) of the 
assistant reference point (32P, 34P, 3 6P) calculated at step 
S708, and "va/' is the corresponding one set of the three sets 
15 of center-coordinates (x a4 , y a4 ) , (x a5 , y a5 ) and (x.-, y a6 ) 

respectively representing the assistant reference points 32P, 
34P and 36P obtained from the picture IMl . 

At step S712, it is determined whether the calculated 
Ae is equal to or smaller than a predetermined permissible value 
20 "E" . Note, the permissible value E may be set as corresponding 
to a dimension of 1/4 pixel. 

If Ae > E, the control proceeds to step S714, in which 
the flag DF is set to "0", thereby indicating that the 
calculation of the camera parameters AX, AY, AZ , a, P and y is 
25 incorrect. 
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On the other hand, if Ae < E, the control returns to 
step S704, and the routine comprising S704, S706, S708 , S710 
and S712 is again executed. Namely, when all the calculated 
coordinates p 4 (x p4 , y p4 ) , p 5 U p5/ Y p5 ) andp 6 (x p6 , y p6 ) respectively 
5 coincide with the center-coordinates (x a4 , y a4 ) , (x a5 , y a5 ) and 
(x a6 , y a6 ) of the assistant reference points 32P, 34P and 36P 
within a permissible range defined by the value "E" (S712) , 
i.e. when the calculation of all the coordinates p 4 <x p4 , y p4 ) , 
p 5 (x p5 , y p5 ) and p s (x p6 , y p6 ) are successively completed, the 

10 control proceeds from step S704 to step S716, in which the flag 
DF is made to be "1", thereby indicating that the calculation 
of the camera parameters AX, AY, AZ , a, P and y is correct. 

In the aforesaid preferred embodiment, the target 20 
is provided with three assistant reference points 32P, 34P and 

15 36P. nevertheless, in another embodiment, only one assistant 
reference point or more than three assistant reference points 
may be arranged on a target to confirm an image position of 
the target . 

Finally, it will be understood by those skilled in the 
20 art that the foregoing description is of preferred embodiments 
of the system, and that various changes and modifications may 
be made to the present invention without departing from the 
spirit and scope thereof. 

The present disclosure relates to subject matter 
25 contained in Japanese Patent Application No. 11-251952 (filed 
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on September 6, 1999) , which is expressly incorporated herein, 
by reference, in its entirety. 
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CLAIMS 



1. An image processing computer system for a 
photogrammetric analytical measurement, in which camera 
5 parameters necessary for production of a survey map are 

determined based on a picture having an image of a target, said 
camera parameters representing a photographing position and 
a photographing direction of a camera, by which said picture 
is photographed, the target having at least three main reference 

10 point areas and at least one assistant reference point area, 
each of the reference point areas being formed as a high 
luminance point area surrounded by a low luminance area, said 
computer system comprising: 

a target-image extractor that extracts the image of said 

15 target from said picture based on positional relationships 
between the main and assistant reference point areas of said 
target; 

an image processor that processes the extracted image 
of said target to determine a two-dimensional position of each 
20 of said main and assistant reference point areas of said target 
with respect to a two-dimensional picture coordinate system 
defined on said image; 

a first calculator that calculates three-dimensional 
positions of said main reference point areas with respect to 
25 a three-dimensional camera coordinate system defined on said 
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camera ; 

a second calculator that calculates two-dimensional 
positions of said main reference point areas with respect to 
a two-dimensional image-plane coordinate system, defined on 
5 an image-plane of said camera, based on the three-dimensional 
positions of said main reference point areas calculated by said 
first calculator; and 

a third calculator that calculates camera parameters 
based on the two-dimensional positions of said main reference 
10 point areas with respect to said two-dimensional picture 

coordinate system and the two-dimensional positions of said 
main reference point areas with respect to said two-dimensional 
image-plane coordinate system. 

2 . An image processing computer system as set forth 
15 in claim 1, further comprising: 

a fourth calculator that calculates a three- 
dimensional position of said assistant reference point area 
with respect to said three-dimensional camera coordinate 
system based on the camera parameters calculated by said third 
20 calculator; 

a fifth calculator that calculates a two-dimensional 
position of said assistant reference point area with respect 
to said two-dimensional image-plane coordinate system based 
on the three-dimensional position of said assistant reference 
25 point area calculated by said fourth calculator; and 



65 



a determiner that determines whether the calculation 
of the camera parameters by said third calculator is correct 
or incorrect by comparing the two-dimensional position 
obtained by said image processor with the two-dimensional 
5 position of said assistant reference point area calculated by 
said fifth calculator. 

3. An image, processing method for a photogrammetric 
analytical measurement, in which camera parameters necessary 
for production of a survey map are determined based on a picture 

10 having an image of a target, said camera parameters representing 
a photographing position and a photographing direction of a 
camera, by which said picture is photographed, the target having 
at least three main reference point areas and at least one 
assistant reference point area, each of the reference point 

15 areas being formed as a high luminance point area surrounded 
by a low luminance area, said method comprising steps of: 

extracting the image of said target from said picture 
based on positional relationships between the main and 
assistant reference point areas of said target; 

20 processing the extracted image of said target to 

determine a two-dimensional position of each of said main and 
assistant reference point areas of said target with respect 
to a two-dimensional picture coordinate system defined on said 
target ; 

25 calculating three-dimensional positions of said main 
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reference point areas with respect to a three-dimensional 
camera coordinate system defined on said camera; 

calculating two-dimensional positions of said main 
reference point areas with respect to a two-dimensional 
5 image-plane coordinate system, defined on an image-plane of 
said camera, based on the three-dimensional positions of said 
main reference point areas ; and 

calculating camera parameters based on the two- 
dimensional positions of said main reference point areas with 
10 respect to said two-dimensional picture coordinate system and 
the two-dimensional positions of said main reference point 
areas with respect to said two-dimensional image-plane 
coordinate system. 

4 . An image processing method as set forth in claim 
15 3, further comprising: 

calculating a three-dimensional position of said 
assistant reference point area with respect to said three- 
dimensional camera coordinate system based on the calculated 
camera parameters ; 
20 calculating a two-dimensional position of said 

assistant reference point area with respect to said two- 
dimensional image-plane coordinate system based on the 
calculated three- dimensional position of said assistant 
reference point area; and 
2 5 determining whether the calculation of the camera 
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parameters is correct or incorrect by comparing the two- 
dimensional position based on said two-dimensional picture 
coordinate system with the two-dimensional position of said 
assistant reference point area based on said two-dimensional 
5 image-plane coordinate system. 

5 . A memory medium storing an image processing program 
for a photogrammetric analytical measurement, in which camera 
parameters necessary for production of a survey map are 
determined based on a picture having an image of a target, said 
10 camera parameters representing a photographing position and 
a photographing direction of a camera, by which said picture 
is photographed, the target having at least three main reference 
point areas and at least one assistant reference point area, 
each of the reference point areas being formed as a high 
15 luminance point area surrounded by a low luminance area, said 
program comprising steps of: 

extracting the image of said target from said picture 
based on positional relationships between the main and 
assistant reference point areas of said target; 
20 processing the extracted image of said target to 

determine a two-dimensional position of each of said main and 
assistant reference point areas of said target with respect 
to a two-dimensional picture coordinate system defined on said 
target; 

25 calculating three-dimensional positions of said main 
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reference point areas with respect to a three-dimensional 
camera coordinate system defined on said camera; 

calculating two-dimensional positions of said main 
reference point areas with respect to a two-dimensional 
5 image-plane coordinate system, defined on an image-plane of 
said camera, based on the three-dimensional positions of said 
main reference point areas ; and 

calculating camera parameters based on the two- 
dimensional positions of said main reference point areas with 
10 respect to said two-dimensional picture coordinate system and 
the two-dimensional positions of said main reference point 
areas with respect to said two-dimensional image-plane 
coordinate system. 

6 . A memory medium as set forth in claim 5 , wherein 
15 said program further comprises : 

calculating a three-dimensional position of said 
assistant reference point area with respect to said three- 
dimensional camera coordinate system based on the calculated 
camera parameters ; 
20 calculating a two-dimensional position of said 

assistant reference point area with respect to said two- 
dimensional image-plane coordinate system based on the 
calculated three-dimensional position of said assistant 
reference point area; and 
25 determining whether the calculation of the camera 
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parameters is correct or incorrect by comparing the two- 
dimensional position based on said two-dimensional picture 
coordinate system with the two-dimensional position of said 
assistant reference point area based on said two-dimensional 
5 image-plane coordinate system. 
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IMAGE -PROCESSING COMPUTER SYSTEM 
FOR PHOTOGRAMMETRIC ANALYTICAL MEASUREMENT 

5 ABSTRACT OF THE DISCLOSURE 

In an image processing computer system for a 
photogrammetric analytical measurement, a picture has an image 
of a target with three main reference point areas and an 

10 assistant reference point area. Positions of the reference 
point areas are determined by a two-dimensional picture 
coordinate system defined on the target. Positions of the main 
reference point areas are calculated with a three-dimensional 
camera coordinate system defined on a camera . Positions of the 

15 main reference point areas are calculated by a two-dimensional 
image-plane coordinate system, defined on an image-plane of 
the camera, based on the calculated three-dimensional 
positions of the main reference point areas . Camera parameters 
are calculated based on the two-dimensional positions of the 

20 main reference point areas of the picture coordinate system 
and the positions of the main reference point areas of the 
image-plane coordinate system. 
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duty to disclose information which is material to patentability as 
defined in Title 37, Code of Federal Regulations §1 .56 which became 
available between the filing date of the prior application and the 
national or PCT international filing date of this application. 



(Application No.) 



(Day/Month/Year Filed) 



(Day/MonthA'ear Filed) 



(Status) 

(patented, pending, abandoned) 



(Status) 

(patented, pending, abandoned) 



1 □ *OfttfD-&«HX!±@IS«p8faiJH#^f±3J««35i1ilflBtlt*»=Tie« 

= -ts. 

l mmm m-xm ^51*43 n&i-m lx. is#B#s±xi±fts 



□ Additional U.S. or international application numbers are listed 
on a supplemental priority sheet attached hereto. 

I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the 
like so made are punishable by fine or imprisonment, or both, under 
Section 1 001 of Title 1 8 of the United States Code and that such 
willful false statements may jeopardize the validity of the application 
or any patent issued thereon. 

The undersigned hereby authorizes the U.S. attorney or agent 
named herein to accept and follow instructions from either his foreign 
patent agent or corporate representative, if any, as to any action to 
be taken in the Patent and Trademark Office regarding this 
application without direct communication between the U.S. attorney 
or agent and the undersigned. In the event of a change in the 
persons from whom instructions may be taken, the U.S. attorney 
or agent named herein will be so notified by the undersigned. 
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Japanese Language Utility or Design Patent Application Declaration 



mn-rz z t &&iz z fum^^-mnn^mmnff^n^x 

Hg#^ 7055 

Neil F. Greenblum 
Bruce H. Bernstein 
Roger P. Glass 
James L. Rowland 
Arnold Turk 



POWER OF ATTORNEY: As a named inventor, I hereby appoint 
the attorney(s) and/or agent(s) associated with the Customer 
Number provided below to prosecute this application and 
transact all business in the Patent and Trademark Office 
connected therewith, and direct that all correspondence be 
addressed to that Customer Number: 

CUSTOMER NUMBER 7055 
The appointed attorneys presently include: 

Reg. No. 28,394 
Reg. No. 29,027 
Reg. No. 30,841 
Reg. No. 32,674 
Reg. No. 33,094 



Address: GREENBLUM & BERNSTEIN, P.L.C. 

1941 ROLAND CLARKE PLACE 
RESTON, VA20L91 



'== S3gHKrjS$Bfc : (£#Ffe & tfKM^^t) Direct Telephone Calls to: {name and telephone number) 



GREENBLUM & BERNSTEIN, P.L.C. 

(703)716-1191 







Full name of sole or first inventor 
Shiqeru WAKASHIRO 






Inventor's signature Date 




' Qjfr 


Residence 
Tokyo, Japan 


mm 


Citizenship 
Japan 






Post Office Address 

c/o ASAHI KOGAKU KOGYO KABUSHIKI KAISHA. 




36-9, Maenocho 2-chome, 
Itabashi-ku. Tokyo, Japan 


*2 0*laI55W*oft« (KS1-S«#) 


Full name of second joint inventor, if any 




Second Inventor's signature Date 


im 


Residence 


mm 


Citizenship 




Post Office Address 







(Wi^t tz&Z tiPM<D&R%%l%l~ttLX £.l5J1£&tt$ (Supply similar information and signature for third and 

fc £tl{ft-t 3 zt Q ) subsequent joint inventors.) 
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